Systems and methods for smart spaces

ABSTRACT

A smart space may be provided by a hub and an artificial intelligence server in communication with the hub. The hub may receive data from at least one smart object in the smart space. The artificial intelligence server may generate clusters of the data received from each of the at least one smart objects. The server may perform processing comprising using a cluster to detect an anomaly in the smart object, identify the smart object, classify the smart object, determine a user behavior, determine a user mood, determine an energy consumption pattern, or create an automated action, or a combination thereof.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/189,076 filed Nov. 13, 2018, which is a continuation of U.S.application Ser. No. 14/939,997 filed Nov. 12, 2015 (now U.S. Pat. No.10,168,677), which claims priority from U.S. Provisional Application No.62/078,337, filed Nov. 11, 2014, the entireties of which areincorporated by reference herein.

BACKGROUND

In a conventional home system, a user can remotely control and managehome appliances via a portable device. Each home appliance is operatedand controlled manually in many cases. Smart spaces may integratecontrol of a variety of home appliances. Smart spaces use integratedwiring technology, network communication technology, securitytechnology, automatic control technology, and audio and video technologyto integrate control of home appliances. Smart spaces networks mayinclude control panels that a person may use to input settings,preferences, and scheduling information that the smart spaces networkuses to provide automated control the various devices, appliances, andsystems in the home. For example, a person may input a desiredtemperature and a schedule indicating when the person is away from home.The home automation system uses this information to control the heating,ventilation, and air conditioning (“HVAC”) system to heat or cool thehome to the desired temperature when the person is home, and to conserveenergy by turning off power-consuming components of the HVAC system whenthe person is away from the home.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a smart space network according to an embodiment of theinvention.

FIG. 2 shows a smart space network according to an embodiment of theinvention.

FIG. 3 shows a smart space network according to an embodiment of theinvention.

FIG. 4 shows a smart space server according to an embodiment of theinvention.

FIG. 5 shows an artificial intelligence (AI) system according to anembodiment of the invention.

FIG. 6 shows a block diagram of a home gateway module according to anembodiment of the invention.

FIGS. 7A-7E show a home gateway module according to an embodiment of theinvention.

FIG. 8 shows a registration process according to an embodiment of theinvention.

FIG. 9 shows a mapping process according to an embodiment of theinvention.

FIG. 10 shows a mapping process according to an embodiment of theinvention.

FIG. 11 shows a learning schedule and AI algorithm according to anembodiment of the invention.

FIGS. 12A-12E show smart objects according to an embodiment of theinvention.

FIG. 13 is a machine learning process according to an embodiment of theinvention.

FIG. 14 is an anomaly detection process according to an embodiment ofthe invention.

FIGS. 15A-15B show data gathered by a system according to an embodimentof the invention.

FIGS. 16A-16B show data gathered by a system according to an embodimentof the invention.

FIG. 17 is a device detection process according to an embodiment of theinvention.

FIG. 18 is a pattern detection process according to an embodiment of theinvention.

FIGS. 19A-19D are energy audit screenshots according to an embodiment ofthe invention.

FIGS. 20A-20TT are app screenshots according to an embodiment of theinvention.

FIG. 21 is a cluster generation process according to an embodiment ofthe invention.

FIG. 22 is an anomaly detection process according to an embodiment ofthe invention.

FIG. 23 is a device detection process according to an embodiment of theinvention.

FIG. 24 is a composite device detection process according to anembodiment of the invention.

FIGS. 25A-25B are an open/close classifier according to an embodiment ofthe invention.

FIG. 26 is a composite smart object classifier according to anembodiment of the invention.

FIG. 27 is an automated action classifier according to an embodiment ofthe invention.

FIG. 28 is a clustering processor according to an embodiment of theinvention.

FIG. 29 is a scene generation process according to an embodiment of theinvention.

FIG. 30 is an audit process according to an embodiment of the invention.

FIG. 31 is a recommendation process according to an embodiment of theinvention.

FIG. 32 is a mood feedback process according to an embodiment of theinvention.

FIGS. 33A-33E are TV user interface screenshots according to anembodiment of the invention.

DETAILED DESCRIPTION OF SEVERAL EMBODIMENTS

The systems and methods described herein may provide and enable smartspaces for home appliance control and/or control of other devices. Thesesystems and methods may utilize advanced data processing and/orartificial intelligence to provide smart spaces systems and methods thatare capable of learning. Additionally, these systems and methods mayintegrate and interconnect devices within existing infrastructure andwired and wireless home automation networks. Some of the featuresdescribed herein may utilize big data systems, machine learning andartificial intelligence algorithms, cloud computing technologies, andcloud services, for example.

Systems and methods described herein may comprise one or more computers.A computer may be any programmable machine or machines capable ofperforming arithmetic and/or logical operations. In some embodiments,computers may comprise processors, memories, data storage devices,and/or other commonly known or novel components. These components may beconnected physically or through network or wireless links. Computers mayalso comprise software which may direct the operations of theaforementioned components. Computers may be referred to with terms thatare commonly used by those of ordinary skill in the relevant arts, suchas servers, PCs, mobile devices, routers, switches, data centers,distributed computers, and other terms. Computers may facilitatecommunications between users and/or other computers, may providedatabases, may perform analysis and/or transformation of data, and/orperform other functions. Those of ordinary skill in the art willappreciate that those terms used herein are interchangeable, and anycomputer capable of performing the described functions may be used. Forexample, though the term “server” may appear in the specification, thedisclosed embodiments are not limited to servers.

In some embodiments, the computers used in the described systems andmethods may be special purpose computers configured specifically forproviding smart spaces. For example, a server may be equipped withspecialized processors, memory, communication components, etc. that areconfigured to work together to perform smart space control, integration,learning, etc., as described in greater detail below.

Computers may be linked to one another via a network or networks. Anetwork may be any plurality of completely or partially interconnectedcomputers wherein some or all of the computers are able to communicatewith one another. It will be understood by those of ordinary skill thatconnections between computers may be wired in some cases (e.g., viaEthernet, coaxial, optical, or other wired connection) or may bewireless (e.g., via Wi-Fi, WiMax, 4G, or other wireless connection).Connections between computers may use any protocols, includingconnection-oriented protocols such as TCP or connectionless protocolssuch as UDP. Any connection through which at least two computers mayexchange data may be the basis of a network.

Examples of systems that may be controlled by the smart spaces systemsand methods described herein may include, but are not limited to, thefollowing: security technology, indoor video intercom, home monitoring,home burglar alarm, home and cell card, household equipment, householdenergy, audio and video technology, centralized distribution of audioand video, background music, HVAC system, lighting systems, alarmsystems, home theater, entertainment systems, other appliances, etc.

FIG. 1 shows a smart space network 10 according to an embodiment of theinvention. A smart space 90 may include systems (described below) thatmay be in communication with a logstore 40 and/or a datastore/memcache20. Log data (e.g., data about smart space 100 usage and trends,discussed below) may be stored in the logstore 40. App data (e.g., inputby a user) may be stored in datastore 20. In some embodiments, thedatastore 20 may be a non-SQL database and realtime data processingtechnologies (e.g., using one or more of Cassandra, BigTable, Dataflow,Redis, MongoDB, and/or other systems). Additionally, app data may beused in translation 50 and/or search 60 functions. Log data and/or appdata may be further stored in cloud storage 30 accessible by thelogstore 40 and datastore 20. Big data queries 70 and/or predictions 80(described below) may be performed by remote servers using the datastored in cloud storage 30 in some embodiments.

FIG. 2 shows a smart space network 100 according to an embodiment of theinvention. Whereas FIG. 1 illustrated a network 10 in terms of functionsthat may be provided, FIG. 2 illustrates a relationship between hardwareelements. For example, a hub 110 and a plurality of peripheral devices120 may be in communication with one another as shown. Each peripheraldevice 120 may be a device controlled by the hub 110 and/or from whichdata is gathered by the hub 110. For example, the peripheral devices 110may be smart appliances and/or devices such as smart plugs or smartsockets that gather data from and/or control appliances. One or moreuser computers may be equipped with apps 132-136 that may allow the userto interact with the hub 110 via a local or wide network 140 (e.g., ahome network or the Internet). The hub 110 may also perform smart TVcontrol features (e.g., streaming media, DVR, etc.) and may display userinterfaces on the TV (e.g., as shown in FIGS. 33A-33E). In someembodiments, the hub 110 and TV may function together to allow a user toperform all smart space functions that may be otherwise performed viaapps 132-136. In effect, the hub 110 may function as a computer, and theTV may function as a display, and the hub 110 may provide app 136 forthe use of the user. Using the TV controls (e.g., a remote control)and/or a remote control provided with the hub 110, the user may interactwith the app 136 via the TV and hub to perform the functions describedherein.

Additionally, external elements such as third party/B2B apps 152/154,third party databases 156, and/or third party ecommerce platforms 158may be in communication with the hub 110 and/or apps 132-136 via thenetwork 140. A system ecommerce platform 160 may also be incommunication with the hub 110 and/or apps 132-136 via the network 140.The system ecommerce platform 160 may include a variety of data (e.g.,external user databases 172, content management systems (CMS) 174,customer relationship managers (CRM) 176). In some embodiments, forexample, the ecommerce platform 160 and/or third party platforms 158 mayallow the user to install applications to display multimedia content,install IoT applications, share social media, and/or add features toreceive predictions and recommendations from the smart home devices andIoT devices.

The apparatus allows the app marketplace to auto install new servicesand applications in background to deliver future new services, contentproviders and control new devices and protocols among other applicationswhich could extend future applications and new services as they becomeavailable

FIG. 3 provides an alternative view of the smart space network 100,illustrating specific protocols, apps, and features that may be includedin the elements of the network 100.

FIG. 4 shows a smart space server 200 according to an embodiment of theinvention. The server 200 may be disposed within the network 140 of FIG.2, for example, and may communicate with the hub 110 and/or apps132-136, for example. The server 200 may include AI and predictionalgorithms/modules, such as a machine learning training module 240, amachine learning (ML) running module 250, and/or a machine learningmonitoring module 260. The server 200 may also include a variety ofdatabases, such as a device (e.g., Internet of Things (IoT)) datastore210, a machine learning training datastore 220, and/or an IoT big datadatastore 230. As described in greater detail below, data gathered atperipheral devices 120 and collected by the hub 110 may be sent to theserver 200 and stored in the IoT datastore 210. Such data may be usedfor training (e.g., passed to the ML training data store 220 and used bythe ML training module 240) and/or analysis (e.g., via the ML runningmodule 250 and/or ML monitoring module 260). Various communicationprotocols (e.g., zigbee, z-wave, WiFi, Bluetooth, etc.) and/orinteraction module with communication board may allow specificcommunications and data streams between devices that use separateprotocols from one another. Thus, devices of a variety of types, brands,configurations, etc. may interact within the smart space (via the hub110 and server 200) and thereby be controlled by the artificialintelligence and machine learning functions described herein. The server200 may be configured to send messages like recommendations, alerts, andnotifications to the apps 132-136 (and/or to the hub 110 for display onthe TV 320). The server 200 may gather interaction feedback with theuser, store the feedback, and use it in posterior analytics to helpretrain the machine learning algorithms of the system.

Systems and methods for providing smart spaces may be capable ofoperating and managing various aspects of residential and/or commercialenvironments. In some embodiments, the system may employ an artificialintelligence (AI) system, for example comprising the hub 110 and/or theML modules 240-260 of the server 200. FIG. 5 shows an AI systemaccording to an embodiment of the invention, illustrating the devicesbeing controlled/monitored and the data processing of the hub 110 andserver 200 (e.g., via the listed APIs and/or others). The system may beconfigured to learn and adapt to different scenarios and user habits. AIsystem may automate control based on a user's lifestyle, appliances'energy management capabilities, and the like. For example, the systemmay learn about the user's interaction with their living space usingdevice and sensor data. Using the collected data, the AI system mayprovide recommendations to the user regarding safety, comfort, energyoptimization, and the like.

Some embodiments may connect a smart space to a cloud based or otherwisenetwork accessible remote system. The remote system may be capable ofmanaging and handling big data. The system may include an operatingsystem, machine learning algorithm, and prediction modules to adapt to auser's preferences. The cloud-based system may provide out-of-home orout-of-office access to the premises and also data for the AI system.

In some embodiments the system may include an integration moduleconfigured to integrate the system, big data architecture, mobiledevices, and communication protocols. Furthermore, the system may allowinteroperability of third party devices, appliances, and the like thusenabling seamless integration of the above. In some embodiments, theintegration module may use open standards for interoperability. Forexample, the open standards may comprise protocols from Home Kit,Thread, Insteon, Zigbee, ZWave, and Wi-Fi, among others. The integrationmodule may provide integration of third party smart spaces systems andautomated residential devices.

In some embodiments, the system may provide a gateway for incorporationof a full suite of smart systems (e.g., IoT devices). Example gatewaydevices may include a modem, a router, a network switch, a voice overinternet protocol (VoIP) for digital signals device, an analog telephonyadapter, or a wireless access point, or combinations of the above. Thegateway may provide a mechanism to connect to different devicesindependent of devices manufacturer, or operating system, or firmware,etc. For example, FIG. 6 illustrates a block diagram of a home gatewaymodule (e.g., hub) 110 in accordance with an embodiment of theinvention. The hub 110 may connect to a TV 320 via HDMI 115 or otherinterface. In some embodiments, the hub 110 may be integrated into theTV 320. In addition, the hub 110 may include elements such as a mainboard/processor 111, power supply 112 a/b, SD card slot 113, USB 114,Ethernet 116, WiFi 117, multiprotocol communications dongle 118, and/orinput device (e.g., mouse, remote control, etc.) 119. The hub 110 maycommunicate with third party home automation devices 310 and/or thenetwork 140. A user may interact with the hub 110 using the input device119 and TV 320 to, for example, control the TV and other devices 120and/or receive information from the hub 110.

In some embodiments, the hub 110 may be connected to a personal computeror other device, and firmware/software to interact with the hub 110 maybe downloaded and installed on the computer to further exploit thepotential of the hub 110. The hub 110 may include indicators and a userinterface. In one embodiment, the software for the hub 110 may provide auser with pre-configured commands. For example, the preconfiguredcommands may be help, version, reset, get state/status of any device,set state of any device, bind, factory reset, network details, bootmode, date time command, and bind set.

In some embodiments, the gateway may be provided using a smart box ordongle. In some embodiments, the system may include an operating systembased on a Google Android platform. A game controller, remote control,and/or mobile application may be used to input commands into the system,for example. In some cases the smart box may be attached to a televisionset, and a user may interact with the system via a television interface,for example. Optionally, the smart box may be a central standalone boxincluding a user interface. The system may work with or withoutInternet, router, and/or Wi-Fi. The system may have a server installedand create a network whereby the devices may communicate with the systemwithout the need of Internet or other separate network, for example. Thesmart box may connect a full suite of interconnected devices andapparatus with the cloud learning system via a built-in multi-protocolarchitecture that may operate and manage various aspects of humanenvironments and interfaces. Additionally, the system may be configuredto receive upgrades, add additional application to support newlyintroduced devices, and the like.

The system may integrate multi-protocol third party devices withintelligent discovery and intelligent mapping. The system hardware maycommunicate with a broad range of devices converging many standards andtechnologies. The communication with third party smart spaces devicesmay be accomplished with the use of a communications dongle which may bemulti-protocol. Communication protocols supported by the dongle mayinclude Wi-Fi, Zigbee, Zwave, Thread, Home Kit and Bluetooth Low Energy.Through the communications dongle, the system may control andcommunicate third party devices.

An example enclosure for the system is shown in FIGS. 7A-7E. The designof the case and materials from which it is made may be selected tooptimize wireless range. FIG. 7A shows the bottom view for the box. FIG.7B shows the front view of the box with indicator lights which maycommunicate system status. FIG. 7C shows the isometric view of the box,FIG. 7D shows the left profile view, and FIG. 7E shows the rear profileview.

The system may include plug and play installation and initializationfeatures. The plug and play functions of the system may be performed byinstalled gateway software. For example, when a user plugs the smart boxin to a power supply a first time, a program stored in system memory mayload drivers, automatically initialize the system to connect to anavailable network, and operationalize different modules. For example,with ZigBee, the drivers may run a network initialization sequence thatmay search for the best ZigBee wireless channels available, create aZigBee network in an identified channel, get network information, storethe information locally, and/or automatically upload the information tothe cloud.

When the system is connected to the Internet it may launch an automaticplug and play registration process. Automatic registration process maybegin when the system connects to a local network with Internetconnectivity. FIG. 8 shows an example registration process according toan embodiment of the invention, wherein a hub 110 may register with aserver 200 and, via mobile app 132/134 and/or web app 136, may beassigned to a smart space. For example app login, see FIG. 20GG-HH.After the system connects to the Internet, it may automatically connectto cloud services and automatically register itself with a registrationserver. In FIG. 8, the hub 110 may register with the server 200. Amobile app 132/134 may direct the device on which the app is installedto connect to the hub 110 and server 200. A web app 136 may direct thedevice on which the app is installed to connect to the server 200 andassociate an account with the hub 110. After registration, the systemmay either be added or not added to a space in the cloud. If the userwants to add the system to a cloud space, the user may login to thecloud using an app. Via the app (e.g., the mobile app 132/134), a usermay assign the hub 110 to a smart space. When login is on the samenetwork as the system, the application may detect the system via UPnPand may give the user the option to assign the system to a defined spacein the cloud. Optionally, if the system is not detected via UPNP, themobile device application may allow and/or require the user to enter thesystem MAC address. The registration process may end upon confirmationof the system addition to the application. For an example interface forspace creation/editing, see FIGS. 20II-20NN.

FIGS. 9 and 10 illustrate example device mapping processes according toembodiments of the invention, wherein the hub 110 may discover devices120 automatically and/or a user, via mobile app 132/134 and/or web app136, may add devices 120. For an example of a user interface for addingdevices 120, see FIG. 20A-20B. For an example of a user interface forediting devices 120, see FIG. 20J-20O. A set of networking protocols maybe installed as part of the system software/firmware that may permitnetworked devices, such as personal computers, printers, Internetgateways, Wi-Fi access points, mobile devices, or any enterprise-classdevices to seamlessly discover each other's presence on the network andestablish functional network services for data sharing, communications,and/or entertainment. For example, the protocol may be a Universal Plugand Play (UPnP) protocol.

In FIG. 9 in 901, a user (via app 132-136) may direct the hub 110 toscan for zwave devices. In 902, devices 120 may respond to the hub 110,and in 903 the hub 110 may request confirmation of the devices 120 fromthe user, or in some cases devices may be set up automatically withoutthe interaction of the user and the hub 110 may scan automatically. Theuser may confirm to the hub 110 that the devices 120 should be added(via app 132-136) in 904. If the add device process is being performedautomatically, the user may be able to delete devices that they don'twant in the network. In 905, the hub 110 may communicate with eachdevice 120 to be added, asking for and receiving basic deviceinformation and manufacturer information. In 906, the hub 110 may reportthe received information to the server 200, which may return the device120 activities to the hub 110 in 907, allowing the hub to control thedevice 120. Additionally, in 908 the server 200 may generate a userinterface for the device 120 and send the user interface to the app132-136 in 909.

In FIG. 10 in 1001, a user (via app 132-136) may direct the hub 110 toscan for ZigBee devices. In 1002, devices 120 may respond to the hub110, and in 1003 the hub 110 may request confirmation of the devices 120from the user, or in some cases devices may be set up automaticallywithout the interaction of the user and the hub 110 may scanautomatically. In 1004, the user may confirm to the hub 110 that thedevices 120 should be added (via app 132-136. If the add device processis being performed automatically, the user may be able to delete devicesthat they don't want in the network. in 1005, the hub 110 maycommunicate with each device 120 to be added, asking for and receivingmanufacturer ID, product ID, active endpoints, active endpoint inputclusters, and active endpoint output clusters. In 1006, he hub 110 mayreport the received information to the server 200, which may return thedevice 120 activities to the hub 110 in 1007, allowing the hub tocontrol the device 120. Additionally, in 1008 the server 200 maygenerate a user interface for the device 120 and send the user interfaceto the app 132-136 in 1009.

The system may automatically start discovering devices in differentnetworks like LAN (through Ethernet or Wi-Fi and different protocolslike upnp/dlna), ZigBee, zwave, thread, homekit, etc. The system mayperform an intelligent discovery and mapping process, whereby the systemmay add devices locally and then push the configuration to the cloud.The configuration and auto-joining and programming may be cloned intothe client's smartphones, tablets, computers, etc.

When the networks needs specific security processes (for example zwavedoor locks) for discovering network devices, the system may establishsecure connections and protocols to perform the discovery/mappingprocess. In lieu of automation, an event (physical button in devices,mobile apps, web apps, etc.) trigger may be required by the system tofinalize the mapping/discovery process.

The intelligent mapping software may discover devices and communicationprotocols from third party vendors to integrate with the system. Thediscovery may be accomplished through the hub 110 and server 200communicating with the devices 120 and checking the received dataagainst known device API data, for example. The intelligent mappingsoftware of the integration module may automatically load the devices'characteristics to the system. Additionally, the integration module mayautomatically back up the data to cloud. Moreover, the intelligentmapping software may generate automatic hardware handlers. These handlesmay be stored and used by different modules to communicate and controlwith these devices.

FIG. 9 illustrates a Zwave Intelligent Mapping Process in accordancewith an embodiment. The process may involve triggering the system toscan devices available in the ZWave network and adding the device to thesystem ZWave network. Once a ZWave compatible device is added to thesame network as the system, the system may get basic information fromthe device.

Next, the system may send the command “get manufacturer information”with the “Node Id”. This may return the manufacturer informationresponse from the network. The process may create a unique identifierfor each new device detected. For example, the unique identifier mayinclude a product type ID and/or a serial number of the device. Theinformation obtained from the Zwave device may include serial number,manufacturer information, and commands supported by the device, forexample. The machine learning algorithms may apply the informationgathered, detect the devices' capabilities, and use the automatic userinterface generation to produce the user interface for the deviceautomatically.

FIG. 10 illustrates a Zigbee Intelligent Mapping Process in accordancewith an embodiment. A trigger from a mobile device and/or webapplication may command the system to scan devices available in theZigbee network and add detected devices to the gateway. The system mayquery manufacturer identity from the ZigBee compatible device. In casethe device is not recognized after acquiring the data, the system maysend a query to the device's active endpoints and receive a list of theactive endpoints from the device. After getting the list of the device'sactive endpoints, the system may send a request to the input clustersand output clusters of each endpoint to get a list of all clusters ineach endpoint. This may provide the device type, the endpoint'sfunctionality, and commands supported by the endpoint. After gatheringthe aforementioned information, machine learning algorithms may beapplied to the information. The machine learning may further detect thedevice's capabilities and may use the automatic user interfacegeneration to produce the user interface for the device automatically.

Once the hub 110 is set up and devices 120 are added, the hub 110 andserver 200 may begin monitoring and control of devices 120 in the smartspace. FIG. 11 shows a learning schedule and AI algorithm according toan embodiment of the invention. As described above, the hub 110 mayfirst be set up, and then the hub 110 and server 200 (collectively “thesystem”) may start to collect data from the devices 120. As data comesin, the system may learn patterns of operation of the devices 120 andtrain the modules 240-260. User-created scenes and rules may be input aswell (e.g., defining when devices 120 may or may not operate and/orcreating “scenes” or “butlers” which may be programmed sequences ofdevice 120 activation and/or deactivation, alone or in combination withother devices 120). For an example interface for creating scenes orbutlers, see FIG. 20D-I. For example interfaces for using created scenesor butlers and/or individual devices, see FIGS. 20P-20FF. As learningtakes place, the system may recognize improvements that may be made tothe smart space's efficiency and offer optimization suggestions (and/ortake action to optimize), including sending alerts to a user (e.g., viaan app 132-136, see FIG. 20C). As time goes on, the system may effectimprovements in smart space efficiency and livability based on the dataobservation and learning.

Some objects controlled by the system may be smart objects. Smartobjects may include data analysis and summarization features to providedata about device type, appliance behavior recognition, usage patternrecognition, anomaly detection, geo-localization usage, automatedaction, and/or other features. The hub 110 and/or server 200 may detectthe kind of appliance connected through the smart device. The hub 110and/or server 200 may perform activity, room / space, and/or geolocationdetection and form clusters of the detected data from the smart object.The hub 110 and/or server 200 may detect device usage patterns over timebased on the data clusters. The hub 110 and/or server 200 may detectdifferent abnormal behaviors related to the data gathered by the smartobject (e.g., the usage pattern data). The smart object may transmitgathered data to the hub 110, and the hub 110 and/or server 200 mayperform usage pattern and/or anomaly detection, for example. The smartobject may also include Automated scenes and butlers generation.

FIGS. 12A-12E show exploded views of smart objects according to anembodiment of the invention. These smart objects are presented asexamples only, and those of ordinary skill in the art will appreciatethat other objects and/or configurations may be possible.

A door/window sensor 1210 may include two units configured to be mountedin proximity to one another (one to the door/window and one to astationary surface nearby) and sense when the door/window isopened/closed. For example, a first unit may include a front plate 1211,a fastener 1212, a circuit 1213, and a housing 1214. The second unit mayinclude a front plate 1215, a magnet 1216, and a rear plate 1217. Thecircuit 1213 may detect the presence/absence of the magnetic field ofthe magnet 1216 and report this detection (e.g., via WiFi, Bluetooth, orsome other connection) to the hub 110. This sensor may be attached tothings that can be opened and closed (e.g., doors, windows, cabinets,drawers, etc.) and may generate alerts when they are opened. Theopen/closed sensor may detect motion with an accelerometer and/or mayinclude sensors for temperature, humidity, and luminosity.

A smart plug 1220 may include an outlet 1221, a terminal 1222, one ormore circuit boards 1223/1224, a housing 1225, a button 1226 that may bein contact with one or more of the circuit boards 1223/1224 and mountedin the housing 1225, and an inlet 1227/1228. The circuit boards1223/1224 may include a circuit that may detect the presence and/orpower usage of a device plugged into the outlet 1221 and report thisdetection (e.g., via WiFi, Bluetooth, or some other connection) to thehub 110. The smart plug may turn any electrical appliance into a smartdevice by simply plugging the appliance into the smart plug. The smartplug may be placed between the power outlet and the device. The smartplug, in conjunction with the system, may allow users to increaseefficiency, turn devices on and off, and/or monitor and control energyconsumption from anywhere. Users may be able to keep track of the energyusage and automatically interrupt the electrical supply if the smartplug detects high temperature in the environment. If the smart plug isplugged into a lamp, it may detect the lumens in the environment andturn the lamp when it gets dark. The smart plug may also dim the lightsof the lamps. The smart plug 1220 may be configured to avoid coveringother outlets (e.g., if the smart plug 1220 is plugged into a firstoutlet in a 2-outlet wall socket, it may not cover the second outlet).

A sensor unit or smart station 1230 may include a top cover 1231/1232, abottom cover 1233, a front cover 1234, a back cover 1235, one or morefasteners 1236/1238, a power inlet 1237, and a circuit 1239. The circuit1239 may include one or more sensors (e.g., light sensors, gas sensors,temperature sensors, etc.). The circuit 1239 may report sensor outputs(e.g., via WiFi, Bluetooth, or some other connection) to the hub 110.For example, the smart station may include a built-in camera and/orother sensors and may measure emotion, face detection, air quality,smoke, CO, temperature, pressure, presence, motion, humidity,luminosity, etc. For example, the smart station may automate lamps toturn on when darkness is detected. The smart station may detect motionusing an accelerometer to remove false positives. The smart station mayalert for fire and may include “Air Quality Detection” (Smart StationProtect) to send alerts and help prevent CO poisoning.

A smart socket 1240 may include a bottom cover 1241, a circuit 1242, atop cover 1243, and a socket thread 1244. The socket thread 1244 maysupply power from the socket into which the smart socket 1240 is screwedto the light screwed into the smart socket 1240. The circuit 1242 maydetect the presence and/or power usage of a device screwed into thesmart socket 1240 and report this detection (e.g., via WiFi, Bluetooth,or some other connection) to the hub 110. The smart socket may allow auser to monitor energy usage, dim the lights, turn them on/off when fromanywhere, set them in random patterns to provide the appearance of acrowded house, etc.

An infrared skipper 1250 may include a top cover 1251, one or morefasteners 1252, a circuit 1253, a middle cover 1254, and a bottom cover1255. The circuit 1253 may include an infrared transceiver that maydetect appliances using infrared remote control (e.g., televisions andother home theater components, etc.). The circuit 1253 may be configuredto control such appliances as well as detect information about theappliances. The circuit 1253 may receive commands from and report datato (e.g., via WiFi, Bluetooth, or some other connection) the hub 110.Infrared is shown in this example, but in some embodiments, the skipper1250 may communicate with appliances via Bluetooth or other connectionin addition to or instead of via infrared. The skipper may function as auniversal remote control and IR blaster. The skipper may be placed in aroom, and it may detect all the appliances that have infrared technology(AC, TV, DVR, and audio system, for example). Using the infraredconnection, the skipper may allow users to control the devices fromanywhere.

The system may also include a presence tag (e.g., an RFID tag that maycommunicate with the hub 110 and/or a smart device such as thosedescribed above) in some embodiments. The presence tag may trigger analert if the tag is removed from within a specified zone. The zone maybe customizable using the system. For example, a child's backpack may betagged, and the tag may trigger an alert when the child is back fromschool. The presence tag may be implemented with key chains, petcollars, etc. The tag may allow the system to know if the tag is withinrange. Optionally, the presence tag may automatically trigger actionsbased on a user's presence. For example, when a user walks into theliving room, the system may play favorite music, turn the lights on, orpreform any other customized function.

The system may also include a smart wall unit that may convert anysocket or switch without having to change the outlet. It may beinstalled inside a box in a wall, and it may be compatible with standardelectrical boxes (e.g., wall sockets or switches). The smart wall unitmay allow on/off switching, dimming, and energy monitoring, among otherfunctions, similar to the smart plug 1220 and smart socket 1240 excepthard-wired within the electrical box.

The system may also be configured to communicate with third partydevices such as smart thermostats, plugs, and dimmers manufactured byCentralite and others; and/or door locks manufactured by Kwikset andothers.

The following is an example set of smart object classification categorytypes that may be used by the system and artificial intelligence:

-   -   Appliance    -   Lighting    -   Doors & Windows    -   Entertainment    -   Other

Within each category, specific smart objects may be provided, forexample as follows:

-   -   Appliances:        -   Coffee maker        -   Toaster        -   Refrigerator        -   Microwave oven        -   Washing machine        -   Dryer        -   Dishwasher        -   Freezer    -   Lighting:        -   Reading lamp        -   Bed lamp        -   Spotlight        -   Ceiling light        -   Chandelier        -   Wall light        -   Outdoor light    -   Doors & Windows:        -   Front door        -   Garage door        -   Outside door        -   Inside door        -   Gate        -   Window    -   Entertainment:        -   Tv        -   Media player        -   Game console        -   Music player        -   Speaker        -   Computer        -   DVR/TiVo    -   Other:        -   Telephone        -   Router        -   Heater        -   Air conditioning        -   HVAC        -   Fan        -   Ceiling fan

The system may learn different aspects of its environment (e.g., usersand devices) and perform smart object classification automatically. Thesystem may include learning algorithms tuned to apply specifically tosmart spaces and commercial systems and internet of things systems andarchitectures. The algorithm may work in conjunction with a big datamodule. The big data module may detect and capture events. For example,if a user manually turns on a switch, the big data module may capturethat event, or if a bulb dies, the big data module may capture thatevent through one or more anomaly detection modules, and so on. Thesystem may be able to learn more efficient ways to save energy. The AImodule can learn from data collected from users, sensors, actuators,etc. and provide cost saving options to home and commercialenvironments. The AI module may further enable users to obtain metricsfrom appliances and provide suggestions on functions the user may notyet be applying.

The system may learn from a user's functions and behaviors and programitself. The data gathered may be updated on a cloud. For example, the AImodule may learn habits, preferences, and schedules for a full range ofappliances continuously. The learning algorithms and cloud event drivenarchitecture may allow auto programming for rules, scenarios, actions,schedules, and triggers to create and send alerts, suggestions,notifications through multi-way channels like SMS, voice call, email,push notifications, etc.

Learning algorithms may also be connected to artificial intelligenceapplication programing interfaces (API)s that may interact withmulti-cloud 3rd party APIs in order to interact with different cloudservices like IFTTT, email servers, etc. Some example artificialintelligence APIs are listed below:

-   ai.datastream.add-   ai.datastream.delete-   ai.datastream.get-   ai.energy.getConsumption-   ai.energy.getSaving-   ai.energy.predictConsumption-   ai.lifestyle.getHabit-   ai.lifestyle.getUserPatterns-   ai. lifestyle. predictScene-   ai.prediction.analyze-   ai.prediction.create-   ai.prediction.delete-   ai.prediction.get-   ai.prediction.list-   ai. prediction. update-   ai.classifyOpenClose-   ai.feedbackAnomalyDetection-   ai.feedbackMoodFeedback-   ai.feedbackOpenCloseClassification-   ai.getBaseline-   ai.getResult-   ai.insertAnomalyDetectionRecommendation-   ai. insertMoodFeedback-   ai.insertOpenCloseClassificationRecommendation-   ai.isAnomaly-   ai.smartobject.getDetectedObj-   ai.smartobject.getDetectedCompoundObj-   ai.smartobject.getHabit-   ai.smartobject.getAutomatedActions-   ai.sendRecommendation-   ai.sendRecommendationV2-   ai.getEnvironment-   ai.getEnvironmentByDevice-   ai.getEnvironmentDetailsByDeviceByHour

The learning features may enable energy saving and energy managementcapabilities for the system. A system may monitor energy consumingdevices, user data, and environmental data collected by sensors, users,and devices such as smartphones, tablets, computers, appliances,electrical devices, etc. The system may analyze the data collected withartificial intelligence algorithms and machine learning algorithms toauto-program a set of one or more actions, rules, scenarios,notifications, suggestions, and/or alerts, and execute different actionsand scenarios to provide strategic reduction of power usage in home,offices (or any space) combined with a multi-sensing, wirelesslycommunicating smart TV and smart TV box home gateway.

The artificial intelligence and learning algorithms may allow efficientlearning for energy management and energy saving features. Differentschedules, scenarios, actions, and/or rules may be created in the systemand may be automatically generated based on immediate-control inputs.Artificial intelligence and machine learning methods may also be usedfor receiving user input relating to the user's preference andenvironments. The energy monitoring and management system may also runin a cloud energy monitoring and energy saving system that may interactwith the system and any electrical device in a location or space.

Each of the functions described below may use machine learning. FIG. 13is a machine learning process 1300 according to an embodiment of theinvention. In 1310, data may be received at the hub 110 from one or moredevices 120. This data may be passed to the server 200 which, in 1320,may build clusters of recent historical data for each device 120. In1330/1340, the system may wait for new data to come in. When it does, in1350 the hub 110 may pass the data to the server 200 which may retrainthe cluster or clusters.

The learning features may provide the system with anomaly detectionfunctionality which may be used to identify abnormal behaviors relatedto energy consumption of devices within a smart space. The system mayapply filters and classifiers to data from the devices monitored in realtime or near real time to detect abnormal behavior (e.g., behavior thatis different from an established pattern of normal behavior). The systemmay notify a user when abnormal behavior is detected.

Data received from smart devices may be processed to aggregate withhistorical data related to the owner, the space, and the device. Theaggregated data may be filtered with a filter, fixed with the lastcoming events for the related device.

The real-time data collected may be used to fit and fix filters andclassifiers. At the start of the life cycle of the system, or each timethan an abnormal behavior is detected, filters and classifiers may bere-trained with the incoming data stream from devices in the smartspace, assuming the data represent the actual normal behavior for themeasured signal of the device.

FIG. 28 is a clustering processor 2800 according to an embodiment of theinvention. The clustering processor 2800 may include a serializedinstances loader 2810, a train/test split maker 2820, a cross validationfold maker 2830, a growing k-means generator 2840, a serialized modelsaver 2850, a cluster performance evaluator 2860, and a text viewer2870. The clustering processor 2800 may be used to perform a variety ofprocesses related to anomaly detection, device detection, user activitydetection, scene discovery/generation, etc., described below.

FIG. 14 is an anomaly detection process 1400 according to an embodimentof the invention. In 1410, the system may use the recently collectedhistorical data to build a classifier based on time series clusters(i.e., clusters of the signal values arranged by time) to find thenormal status of the behavior of the device based on the incoming signal(FIG. 13). When clusters are determined, in 1420, the distance of anynew event (detected by the hub 110) to the nearest centroid may bedetermined by the server 200. That is, in this case the filters may beintervals around centroids of each cluster. In 1430/1440, if thedistance is normal (e.g., below a threshold), the data may be added tothe cluster. In 1450, an abnormal behavior may be detected if there aremany successive (or near in time) events out of the filter (i.e., awayfrom a centroid).

A specific example of a cluster generation process 2100 for anomalydetection according to an embodiment of the invention is shown in FIG.21. In 2110, a serialized instances loader may accept the historicaldata and output a data set. In 2120, a train/test split maker mayreceive and process the data set and output a training set. In 2130, across validation fold maker may receive and process the training set andoutput a training set and a test set. In 2140, a growing K-meansalgorithm may use the training set and test set from the crossvalidation fold maker to generate the clusters. In 2150, the clustersmay be saved.

FIG. 22 is an anomaly detection process 2200 according to an embodimentof the invention, wherein the clusters generated by the process 2100 ofFIG. 21 may be used to identify anomalies. In 2210, the saved data fromstep 2150 may be loaded, and in 2220, the cluster memberships may bedetermined and output as a data set. Cluster classes may be assigned forthe data set in 2230. In 2240, a train/test split maker may receive andprocess the data set and output a training set. In 2250, a crossvalidation fold maker may receive and process the training set andoutput a training set and a test set. In 2260, a serialized classifiermay classify data against the outputs to identify anomalies. In 2270,the anomaly data may be saved.

In one anomaly detection example, the smart object may be a light with adimmer and/or on-off control (e.g., the smart socket described above ora light with its own smart communications and data gathering features).A filter may be set for the device based on observed historical devicedata. In the case of a dimmer, clusters may be formed around frequentlyobserved dimmer settings. For example, a user may set the dimmer to 20%when they come home from work. In this case the filter may be a simpleinterval around the average of the historical data stream in a timewindow. (e.g., quartiles 2 and 10 of 11 partitions, over a window timedata stream).

The baseline for the smart object may be calculated as range of minimaland maximal samples, for example taking the lowest and highest secondquantiles of 11 buckets. If the number of samples covered in the daterange is less than some threshold (MIN_REQ_SAMPLE_BASELINE), thebaseline may be discarded.

To determine whether a light device has an abnormal behavior, the systemmay take the last samples (from the last 24 hours or at least the lastMIN_REQ_POWER_SAMPLES), count how many samples are out of the baselinerange, and count how many samples are after the first anomaly. If thepercentage of abnormal samples on the last samples is over somethreshold (ANOMALY_TOL_LIMIT), an anomaly may be registered by thesystem.

Similar monitoring may be performed for smart plugs and outlets andother energy-using devices, for example based on energy use sampleswherein when a percentage of abnormal samples on the last samples isover some threshold (ANOMALY_TOL_LIMIT), an anomaly may be registered bythe system. In addition, the type of device plugged into a smart plugmay change. Hence when an abnormal behavior is detected it may indicatethat a real abnormal behavior is taking place or that the applianceconnected was changed. In the latter case, the system may notify theuser of the change but, over time, may learn the new behavior patternand thus recognize that a new device has been connected and startlooking for anomalies with that new device.

The system may detect anomalies in user behavior. The system mayregister actions executed by any user with any client through the IoTcloud to any device. The system may periodically look for anomalies inuser behavior, through actions or occurrences/omissions of events, onthe devices. For each kind of device, the system may store a collectionof typical abnormal behavior patterns, codified as rules, against whichcurrently monitored behavior may be evaluated. When an abnormal userbehavior is detected, depending on the severity of the situation, thesystem might perform different actions, such as notify the user aboutthe issue or request a secure validation of the action, denying accessto the device if secure validation is not received.

The system may detect anomalies in data streaming patterns. This is ageneralization of the energy anomaly detection functionality. The systemmay process the recently historical data and build a classifier based onclusters for time series to find the normal status of the behavior of asignal. For example, data from a smart station (e.g., temperature,humidity, luminosity, and/or activity/motion frequency) may be monitoredfor departures from frequently observed measurements.

The system may periodically look for anomalies in the movements of usersattempting to access the IoT cloud from different geo-locations. Eachtime the system receives a user command from a remote location (e.g.,from a user editing smart space settings in her home while she is at theoffice), the data sent from the device including the command may alsoinclude a GPS location of the device. The system may place a point onthe map corresponding to a GPS location of the device sending thecommand. After several commands are received from a user, the system maygenerate a cluster of points over the map. Then, each time a new commandis received and thus a new point appears, the system may measure thedistance to the nearest centroid. An abnormal geolocation may bedetected when this distance exceeds some limit.

When an abnormal user geolocation is detected, the system may notify theuser about the abnormality and request a secure login, and if the securelogin is not received, access to the smart space controls from theabnormally located device/account may be denied.

In one specific example, the system may use data collected by a bathroomlight switch and a temperature sensor to detect average shower lengthand shower frequency for a user. FIGS. 15A-15B show an example of datagathered illustrating a shower use pattern. For example, if the systemdetermines that a user takes long showers, it may generate a messagerecommending a shorter shower duration including a display of potentialenergy savings associated with reducing shower time.

In another example, the system may use data collected by a smart lightswitch equipped with a power consumption reporting module to detectmalfunctioning lights. For example, if the system determines that alight is malfunctioning, it may generate an alert to facility managementnotifying that a light bulb needs to be replaced. FIG. 16A shows anexample of normal light behavior, and FIG. 16B shows an example ofabnormal light behavior as compared with the normal light behavior ofFIG. 16A.

The system may create an environment that is operating systemindependent. This may allow open APIs for smart spaces and commercialsystems and IoTs to utilize the system's capabilities to provide afriendly environment for developers. With open APIs for software andhardware, developers may be able to integrate third party software andhardware to the system. Thus, devices 120 may be preprogrammed with theability to communicate with the hub 110 in some cases. However, in othercases, the system may perform appliance detection automatically.

Smart devices may be used in combination with a variety of appliances toperform appliance detection. For example, a smart plug may be installedin a smart space, and an appliance which may or may not have smart IoTfeatures may be plugged into the smart plug. The smart plug may be ableto determine the type of appliance, even if the appliance itself doesnot have any smart features.

The smart plug may include a smart object module configured to collectdata about the behavior of the events from the device. Based on thesignals of the device such as energy consumption, number of on/offevents, execution regimen time, and others, the smart object module maylearn which appliance is connected. The smart object module may alsomeasure the correlation of events between many different devices (otherdevice types such as open/close, motion sensor, etc.) from the sameuser, or in the same network, and determine if the devices are in thesame room or if the devices are working together as a composite smartobject.

FIG. 17 is a device detection process 1700 according to an embodiment ofthe invention. Similar to the anomaly detection described above, in 1710the appliance detection may use clustering on time series on real timeor near real time data to identify different statuses of the smartobject in time. In 1720 the appliance detection may use pattern matchingand linear regression over the data for each status to characterize thewave of each status. Based on the identified pattern, in 1730 the systemmay identify the device (e.g., a toaster may have one pattern, while ablender has a different pattern). The system may also measure thecorrelation of the events between many different devices (other devicetypes such as open/close, motion sensor, etc.) from the same user or inthe same network and determine if these devices are in the same roomand/or if these devices are working together as a composite smartobject.

FIG. 23 is a specific example of a device detection process 2300according to an embodiment of the invention. In 2310, a serializedinstances loader may accept the historical data and output a data set.In 2320, a wavelet filter may receive and process the data set andoutput a data set including a wave for the input data. In 2330, clustermembership for the wave may be determined. In 2340, a class may beassigned to the data based on the wave and cluster membership. In 2350,a train/test split maker may receive and process the classified data andoutput a training set. In 2360, a cross validation fold maker mayreceive and process the training set and output a training set and atest set. In 2370, a serialized classifier may output a deviceidentification.

For example, an open/close classification may be a classifier specificfor open/close devices and may run as a batch process over all thedevices that have enough accumulated data and are not tagged as aspecific smart object yet. A sample classifier 2500 is shown in FIGS.25A-25B. The classifier may use detected features (difference betweenopen and close time, the average of the temperature sensed in thedevice, the amount of events along the day, and others) to classify thedevice from among the different possible door/window classes. Once adetermination has been made, the system may automatically tag the deviceand/or notify the user suggesting to tag the device with the detecteddoor/window class. In one example, a door/window sensor may detect afront door based on a pattern of when the door is opened generated asdescribed above. For example, if the door is used daily at certain timesof day (e.g., morning and evening), the detected door openings andclosings may reveal a pattern consistent with a resident coming andgoing from a home, suggesting a front door. The system may automaticallycreate and enable entrance light activation triggers at the time of auser's typical arrival based on this detection, for example. The systemmay change behavior based on detected anomalies. For example, the systemmay determine a user is away from home based on the door not being usedfor a period of time and may temporarily disable the entrance lighttrigger.

In some cases, the system may automatically detect composite smartobjects. FIG. 26 is an example composite smart object classifier 2600according to an embodiment of the invention. For example, the system mayidentify correlations in the activities of different devices during thesame time period to determine if a relationship between the devices mayexist (e.g., TV and receiver are always on at the same time). If suchcombinations are detected, the system may recommend to the user to makea “composite smart object” with the related devices or may automaticallygenerate the composite smart object.

FIG. 24 is a composite device detection process 2400 according to anembodiment of the invention. In 2410, electronic footprints for devices(e.g., as generated in the process 2300 of FIG. 23) may be retrieved. In2420, a class may be assigned to the retrieved data. In 2430, atrain/test split maker may receive and process the classified data andoutput a training set. In 2440, a cross validation fold maker mayreceive and process the training set and output a training set and atest set. In 2450, a relationship tree may be generated to definerelationships among devices. In 2460, a performance evaluator maycorrelate activities/uses of the related devices. In 2470 and 2480,outputs illustrating these relationships in a chart and in text may beoutput, respectively.

FIG. 18 is a pattern detection process 1800 according to an embodimentof the invention. In addition to automatically detecting anomalies anddevices via the clustering, the system may also detect user patterns byperforming similar processing (e.g., in 1810 identifying clusters). In1820, this functionality may look for clusters of events on devices thatpersist in time (e.g., for weeks) and may be scheduled (e.g., eventsthat are executed some days at about some specific time). Then in 1830,the system may take some action in response. For example, the system mayrecommend to the user to set a schedule for this action at this specifictime. The system may also send notifications in cases where events thattypically happen at some time do not happen. Automated actionsfunctionality may include a planning system, a grammar inductionprocess, a ranking algorithm, and a notification manager system todetect the situation and rules that trigger some action or event. FIG.27 is an automated action classifier 2700 according to an embodiment ofthe invention. Each time series translator 2710 may add a set ofselected features from a previous event instance. The result may includea time window instance with features from the current event and (in thisexample) three previous events.

In some cases, the user patterns may be used to generate automatedscenes. Thus, if a user always performs some combination of actions nearthe same time each day, the clustering processing described above maydetect this, and the system may automatically generate commands to causethe actions to be performed automatically. For example, if a user alwaysturns on the lights and the radio when they arrive home from work at ornear 7 PM each weekday, the system may generate an automated scene thatmay cause the hub 110 to turn on the lights and the radio at 7 PM eachweekday without user input. FIG. 29 is a scene generation process 2900according to an embodiment of the invention. User pattern data and/orsmart object pattern data may be received by the system in 2910, and in2920 the system may identify possible scenes associated with the data(e.g., based on frequently observed clustered activities). Candidatesmay be ranked by the system in 2930, and in 2940 the system mayrecommend scenes based on the rankings. In 2950, the system may presentrecommendations to the user for approval. If approved, a scene may berun automatically in the future.

The system may also use clustering to perform an energy audit. FIG. 30is an audit process 3000 according to an embodiment of the invention. In3010, a CSV loader may load energy data. In 3020, classes may beassigned to the data. In 3030, a train/test split maker may process theclassified data, and in 3040, a cross validation fold maker may processthe data. A linear regression may be performed in 3050. In 3060, aclassifier performance evaluator may evaluate the energy performancebased on the linear regression. In 3070 and 3080, text data and aperformance chart may be output, respectively. For example, the systemmay examine data from smart object behaviors (e.g., in door/window smartobjects, temperature sensor, and thermostat) to determine the thermalisolation of the space and the best time to condition the environment.To do this, the system may use linear regression techniques and considerthe weather forecast. For example, weather forecasts may be obtained bythe hub 110 and/or server 200 through external APIs like Accuweather andWeather Underground, among others, and/or the hub 110 and/or server 200may consider real time weather through data shared by external devicesof the current location (e.g., weather stations in the smart space) ornearby locations.

Based on these determinations, the system may recommend when tocondition the atmosphere (using energy for heating or cooling andventilation), and when to use natural ventilation, according to theweather forecast and reading the various sensors from home. Smartobjects (e.g., thermostat) may be scheduled to work at the optimal timesbased on the energy audit analysis to achieve improved energyefficiency. In some cases, this may be combined with hard rules. In oneexample, detection of door/window smart object with status open whilethe thermostat is on and cooling or heating may trigger a notificationand, depending of the configuration, turn off the thermostat. In anotherexample, if it is raining and the automatic sprinkler is working, thesystem may to turn of the sprinkler and/or notify the user. FIG. 31 is arecommendation process 3100 according to an embodiment of the invention.In 3105, an energy audit analysis outcome data set may be loaded. In3110, classes may be assigned to the data. In 3115, a cross validationfold maker may process the data. Bayesian processing may be performed in3120. In 3125, output of the Bayesian processing may be generated. In3130, a classifier performance evaluator may evaluate the energyperformance based on the Bayesian data. In 3140, 3145, and 3150, acost-benefit analysis, model performance chart, and text report may beoutput, respectively. In 3135, a prediction appender may predict whenconditioning may be best performed. In 3155 and 3160, scatter plotmatrices defining when conditioning may be performed may be output.

The system may summarize the environment based on collected data. Theinformation from some or all sensors in the smart space may be collectedand summarized it a data structure useful to data analytics and may bedisplayed to a user. Example screenshots of the data presentations areshown in FIGS. 19A-19D.

In some embodiments, the system may use real time or near real timeinformation collected about the user (e.g., coming from user pattern,geolocation, smart objects, voice recognition, social networksinteractions, and other sources) to perform an interpretation about themood of the user. According to the results, the system may performdifferent actions such as suggesting activities and/or adjusting theenvironment. FIG. 32 is a mood feedback process according to anembodiment of the invention. In 3210, a CSV loader may load speechrecognition or other mood indicative data. In 3220, classes may beassigned to the data. In some cases, classes may be user-assigned (e.g.,the user may input their own mood (not shown)). In 3230, a train/testsplit maker may process the classified data, and in 3240, a crossvalidation fold maker may process the data. A J48 analysis may beperformed in 3250 to extract mood from the data. In 3260, a mood graphmay be output. In 3270, a classifier performance evaluator may evaluatethe mood based on the outcome of the analysis in 3250. In 3280 and 3290,text data and a performance chart may be output, respectively. Theoutput may also control elements in the smart space. As an example,voice recognition input may be processed for sentiment analysis (e.g.,the system may transform voice commands into text words or phrases thatgo into artificial intelligence and machine learning algorithms and areprocessed to detect the mood of the user, where a phrase like “Just hada fabulous day!” means “excited,” etc.). According to the results, thesystem may perform different actions such as suggesting activitiesand/or adjusting the environment (e.g., setting the colors of livinglights to green and blue, turning on the speakers and playing music theuser listens in that mood, opening windows, and turning off thethermostat).

The system may give users a two dimensional (e.g., floor plan) and/orthree dimensional (e.g., 3D model generated by, for example, Away 3D,Paper Vision, and/or WebGL) virtual representation for devices and smartobjects within their environment. The system may create locally andremotely virtual representations of smart objects. Representations maybe detected using the data generated above (e.g., by looking at signalstrengths between devices and the hub 110, power use of devices, audiovolume, temperature, etc.) Representing the smart objects in a virtualscenario may allow the system to create intelligent agents which canself-create automated planning and scheduling of events andnotifications with little user interaction (e.g., presentation forapproval and drag and drop interaction). The intelligent agents may usemachine learning and artificial intelligence algorithms to teach thesystem how the smart objects are used and may continuously learn userpreferences. Non-smart objects like lamps, lights, or plugs may beturned into smart objects with accessories designed to turn them intosmart objects as discussed above, and thus may be virtually representedin the system as well. Objects represented in the system may form a gridof interconnected devices which form a network of ubiquitous computing,sending information to the machine learning algorithms to better learnuser preferences.

The system may optionally provide a user interface factory (UIF)software to automatically generate custom user interfaces. The UIF mayuse the plug and play installation/configuration architecture along withintelligent discovery, mapping, and/or learning algorithms to generatecustom user interfaces for devices. For example, a new or/and unknowndevice may trigger an event that may provide automatic commands to thesoftware to detect a device's features and automatically generate a UIfor the device. This may allow the system to control any device or brandwithout the intervention of new software to support new devices. Forexample, when a Z-wave device is discovered, intelligent mapping mayread the command classes (or clusters in zigbee) and generate a userinterface that contains widgets according to features and capabilitiesof the command classes discovered. The generated UI may feed back to thelearning algorithms and AI module. The AI module may capture theunknown/new device user interactions and preferences and may createimprovements to the user interface. The UI generated by the UI factorymay be operating system independent.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and detail can be made thereinwithout departing from the spirit and scope. In fact, after reading theabove description, it will be apparent to one skilled in the relevantart(s) how to implement alternative embodiments.

In addition, it should be understood that any figures that highlight thefunctionality and advantages are presented for example purposes only.The disclosed methodologies and systems are each sufficiently flexibleand configurable such that they may be utilized in ways other than thatshown.

Although the term “at least one” may often be used in the specification,claims and drawings, the terms “a”, “an”, “the”, “said”, etc. alsosignify “at least one” or “the at least one” in the specification,claims, and drawings.

Finally, it is the applicant's intent that only claims that include theexpress language “means for” or “step for” be interpreted under 35U.S.C. 112(f). Claims that do not expressly include the phrase “meansfor” or “step for” are not to be interpreted under 35 U.S.C. 112(f).

1. A system for providing a smart space, comprising: an artificialintelligence server configured to: receive data from at least one smartobject in the smart space; generate, by applying at least one artificialintelligence algorithm, machine learning algorithm, deep learningalgorithm, or a combination thereof, clusters of the data received fromeach of the at least one smart objects, each cluster comprising datareceived from one of the at least one smart objects during one of aplurality of time periods, wherein a plurality of clusters are generatedfor each of the at least one smart objects, each of the plurality ofclusters for each of the at least one smart objects comprising datareceived during a different time period; perform processing comprisingusing the clusters to classify the at least one smart object, theprocessing comprising: evaluating a plurality of the clusters toidentify a consistency between the clusters, analyzing the plurality ofclusters having the consistency and, based on the analyzing, identifyinga pattern indicative of an object type, and classifying the at least onesmart object as the object type indicated by the identified pattern;provide a user interface including at least one virtual representationof the at least one smart object based on the object type as classifiedby the processing, the user interface being configured to receive a userinput indicating a function specific to the object type; and generateand send a command configured to control the function indicated by thecommand, thereby causing a change in an operation of the at least onesmart object.
 2. The system of claim 1, wherein the user interfacecomprises a drag and drop user interface, and the at least one visualrepresentation is configured to be dragged and dropped within the userinterface to define the command.
 3. The system of claim 1, whereinproviding the user interface comprises identifying, based on theprocessing, at least one feature of the at least one smart object andproviding at least one user interface element corresponding to the atleast one feature.
 4. The system of claim 1, wherein: the user interfaceincludes at least one intelligent agent comprising an artificialintelligence and/or machine learning component, and generating thecommand is further based on code automatically generated by the at leastone intelligent agent.
 5. The system of claim 4, wherein the at leastone intelligent agent automatically generates the code in response to auser preference identified through user interaction with the userinterface.
 6. The system of claim 1, wherein the server is furtherconfigured to: generate analytics information from the data receivedfrom the at least one smart object; and provide the analyticsinformation in the user interface.
 7. The system of claim 1, whereinproviding the user interface comprises sending data enabling display ofthe user interface to a mobile device, a personal computer, atelevision, or a combination thereof.
 8. The system of claim 1, whereingenerating the command comprises identifying a plurality of commandclasses assigned to the at least one smart object according to theobject type and selecting at least one of the command classes as thecommand.
 9. The system of claim 1, wherein the selecting is based atleast in part on intelligent mapping according to the object type. 10.The system of claim 1, wherein the machine learning algorithm comprisesa K-means algorithm.
 11. The system of claim 1, wherein classifying thesmart object comprises: analyzing the clusters to identify at least oneadditional pattern indicative of a plurality of object types, therebyidentifying a plurality of smart objects; and associating the pluralityof smart objects with one another into a composite object.
 12. Thesystem of claim 1, wherein classifying the at least one smart objectincludes determining an energy consumption pattern of the at least onesmart object.
 13. The system of claim 12, wherein determining the energyconsumption pattern comprises identifying events in the datacorresponding to energy use and compiling energy use data for a periodof time.
 14. The system of claim 13, wherein determining the energyconsumption pattern further comprises: obtaining weather information;and correlating the weather information with the energy use.
 15. Thesystem of claim 1, wherein the classifying the at least one smart objectincludes determining a user interaction with the at least one smartobject.
 16. The system of claim 15, wherein determining the userinteraction comprises analyzing the clusters to identify a patternindicative of a repeated user action.
 17. The system of claim 1, whereinthe at least one smart object comprises a door/window sensor, a smartplug, a sensor unit, a smart socket, a skipper, a presence tag, a smartwall unit, a thermostat, a plug, a dimmer, a television, a home theatercomponent, an appliance, a lock, a machine, or a device, or acombination thereof.
 18. The system of claim 1, wherein the commandcomprises an automated action.
 19. The system of claim 18, wherein theautomated action comprises generating an alert.
 20. The system of claim1, wherein generating clusters comprises: obtaining the data over aperiod of time; identifying temporal relationships between events in thedata; and forming the clusters at times indicative of the temporalrelationships.
 21. The system of claim 1, wherein the artificialintelligence server is further configured to perform processingcomprising associating the at least one smart object into a smart spacenetwork.
 22. The system of claim 1, further comprising a hub configuredto receive the data from the at least one smart object and send the datato the artificial intelligence server.
 23. The system of claim 22,wherein at least a portion of the artificial intelligence server and thehub are elements of a combined system.
 24. The system of claim 22,wherein the hub is further configured to control output displayed on amobile device, a personal computer, a television, or a combinationthereof.
 25. The system of claim 22, wherein the artificial intelligenceserver is further configured to install software on the hub.
 26. Thesystem of claim 22, wherein: the hub is in communication with a displayand a controller, and the hub is further configured to provide a userinterface for control of the smart space via the display and receive auser command via the controller.
 27. The system of claim 26, wherein thedisplay is a mobile device, a personal computer, a television, or acombination thereof.
 28. The system of claim 1, wherein classifying theat least one smart object includes identifying an anomaly exhibited bythe at least one smart object based on the evaluating.
 29. A method forproviding a smart space, comprising: receiving, by an artificialintelligence server, data from at least one smart object in the smartspace; generating, by the artificial intelligence server applying atleast one artificial intelligence algorithm, machine learning algorithm,deep learning algorithm, or a combination thereof, clusters of the datareceived from each of the at least one smart objects, each clustercomprising data received from one of the at least one smart objectsduring one of a plurality of time periods, wherein a plurality ofclusters are generated for each of the at least one smart objects, eachof the plurality of clusters for each of the at least one smart objectscomprising data received during a different time period; performing, bythe artificial intelligence server, processing comprising using theclusters to classify the at least one smart object, the processingcomprising: evaluating a plurality of the clusters to identify aconsistency between the clusters, analyzing the plurality of clustershaving the consistency and, based on the analyzing, identifying apattern indicative of an object type, and classifying the at least onesmart object as the object type indicated by the identified pattern;providing a user interface including at least one virtual representationof the at least one smart object based on the object type as classifiedby the processing, the user interface being configured to receive a userinput indicating a function specific to the object type; and generatingand sending, by the artificial intelligence server, a command configuredto control the function indicated by the command, thereby causing achange in an operation of the at least one smart object.
 30. The methodof claim 29, wherein the user interface comprises a drag and drop userinterface, and the at least one visual representation is configured tobe dragged and dropped within the user interface to define the command.31. The method of claim 29, wherein providing the user interfacecomprises identifying, based on the processing, at least one feature ofthe at least one smart object and providing at least one user interfaceelement corresponding to the at least one feature.
 32. The method ofclaim 29, wherein: the user interface includes at least one intelligentagent comprising an artificial intelligence and/or machine learningcomponent, and generating the command is further based on codeautomatically generated by the at least one intelligent agent.
 33. Themethod of claim 32, wherein the at least one intelligent agentautomatically generates the code in response to a user preferenceidentified through user interaction with the user interface.
 34. Themethod of claim 29, further comprising: generating, by the artificialintelligence server, analytics information from the data received fromthe at least one smart object; and providing the analytics informationin the user interface.
 35. The method of claim 29, wherein providing theuser interface comprises sending, by the artificial intelligence server,data enabling display of the user interface to a mobile device, apersonal computer, a television, or a combination thereof.
 36. Themethod of claim 29, wherein generating the command comprises identifyinga plurality of command classes assigned to the at least one smart objectaccording to the object type and selecting at least one of the commandclasses as the command
 37. The method of claim 29, wherein the selectingis based at least in part on intelligent mapping according to the objecttype.
 38. The method of claim 29, wherein the machine learning algorithmcomprises a K-means algorithm.
 39. The method of claim 29, whereinclassifying the smart object comprises: analyzing the clusters toidentify at least one additional pattern indicative of a plurality ofobject types, thereby identifying a plurality of smart objects; andassociating the plurality of smart objects with one another into acomposite object.
 40. The method of claim 29, wherein classifying the atleast one smart object includes determining an energy consumptionpattern of the at least one smart object.
 41. The method of claim 40,wherein determining the energy consumption pattern comprises identifyingevents in the data corresponding to energy use and compiling energy usedata for a period of time.
 42. The method of claim 41, whereindetermining the energy consumption pattern further comprises: obtainingweather information; and correlating the weather information with theenergy use.
 43. The method of claim 29, wherein the classifying the atleast one smart object includes determining a user interaction with theat least one smart object.
 44. The method of claim 29, whereindetermining the user interaction comprises analyzing the clusters toidentify a pattern indicative of a repeated user action.
 45. The methodof claim 29, wherein the at least one smart object comprises adoor/window sensor, a smart plug, a sensor unit, a smart socket, askipper, a presence tag, a smart wall unit, a thermostat, a plug, adimmer, a television, a home theater component, an appliance, a lock, amachine, or a device, or a combination thereof.
 46. The method of claim29, wherein the command comprises an automated action.
 47. The method ofclaim 46, wherein the automated action comprises generating an alert.48. The method of claim 29, wherein generating clusters comprises:obtaining the data over a period of time; identifying temporalrelationships between events in the data; and forming the clusters attimes indicative of the temporal relationships.
 49. The method of claim29, further comprising performing, by the artificial intelligenceserver, processing comprising associating the at least one smart objectinto a smart space network.
 50. The method of claim 29, furthercomprising: receiving, by a hub, the data from the at least one smartobject; and sending, by the hub, the data to the artificial intelligenceserver.
 51. The method of claim 50, wherein at least a portion of theartificial intelligence server and the hub are elements of a combinedsystem.
 52. The method of claim 50, further comprising controlling, bythe hub, output displayed on a mobile device, a personal computer, atelevision, or a combination thereof.
 53. The method of claim 50,further comprising installing, by the artificial intelligence server,software on the hub.
 54. The method of claim 50, wherein the hub is incommunication with a display and a controller, the method furthercomprising providing, by the hub, a user interface for control of thesmart space via the display and receive a user command via thecontroller.
 55. The method of claim 54, wherein the display is a mobiledevice, a personal computer, a television, or a combination thereof. 56.The method of claim 29, wherein classifying the at least one smartobject includes identifying an anomaly exhibited by the at least onesmart object based on the evaluating.